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Prefatory Note 



The research reported in this paper was performed 
by HumRRO Division No. 6 (Aviation), Fort Rucker* 
Alabama, under Work Unit PREDICT, Prediction of 
Training and Operational Performance of Army Aviators . 
The paper was presented at the 1973 Alabama Psycho- 
logical Association meeting held in Destin, Florida, May 
1970. 

Reference is made herein to a paper given by Wiley 
R. Boyles and James L. Wahlbcrg titled, Prediction of 
Army Aviator Performance: Description of a Developing 
System, which preceded the present paper on the pro- 
gram, and is now in preparation as a HumRRO Profes- 
sional Paper, The two papers constitute a portion of the 
current research on prediction of aviator success con- 
ducted by HumRRO Division No. 6. 
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PEER RATINGS AS PREDICTORS OF SUCCESS IN MILITARY AVIATION 



James L. Wahlberg, Wiley R. Boyles, 
and H, Alton Boyd 



Peer evaluations, in the forms of ratings, rankings, and nominations, have been used 
to predict performance in many academic, business, and mihtary settings. While this is 
not intended as a literature review, several examples will illustrate the uses of this 
technique. 

Weitz (1) reported a validity coefficient of ,40 between peer nominations among life 
insurance agents and later success in supervisory positions. He concluded that peers can 
identify differentiating characteristics for supervisory personnel at the time they are all in 
subordinate positions. 

Rodman (2) utilized a peer rating in an industrial setting to identify future top 
executives. Waters and Waters (3) have recently described a small but significant relation- 
ship (r - .31) between peer nominations and performance of salesmen. 

The U.S. military services have had a great deal of success in using peer ratings as 
predictors of military performance, McClure, Tupes, and Dailey (4) concluded that there 
is a higher relationship between peer ratings in Officer Candidate School (OCS) and later 
on-the-job proficiency ratings than between OCS academic or military grades and profi- 
ciency ratings. 

Hollander (6) discussed the use of four foirn3 of peer ratings to predict officer 
performance in the Navy. Although all forms afforded substantial and significant predic- 
tion, the “success as a future officer" nominetion form was the best predictor of iuture 
officer performance when correlated with officer fitness reports covering three years oi 
more of duty (r = .40). In Hollander's study, the third training week peer nomination 
validity was as high as that resulting from the 12th training week administration. 

Medland (6), in studying identification of potential noncommissioned officer (NCO) 
leaders, found that pair ratings (r« .80) were more reliable than cadre ratings (r = .67) in 
either intact or reorganized groups. 

Recently the Army incorporated a "Contemporary Evaluation Form" (CEF) into the 
training program for Aviation Warrant Officer Candidates at the U.S. Army Helicopter 
School (USAHS). The CEF is a USAHS-produced device on which all trainees are 
required to evaluate their squadmates on the basis of potential officer qualities. It is then 
used by training administrators in the counseling and further evaluation of trainees. 
Concurrently, for reasons to be described, HumRRO developed three experimental peer 
rating forms for use in research in prediction of the aviation training performance 
criterion— completion/attrition from the training program. 

The purpose of this paper is to describe the construction of the three HumRRO 
peer ratings, termed the "Potential Aviator Rating" (PAR Forms 1, 2, and 3), and to 
compare the validity of these forms with the CEF used by the U.S. Army Helicopter 
School. 

The subjects were 108 Army Aviation Warrant Officer Candidates in Rotary Wing 
flight training class 1970-23. The students in this class were all volunteers, as are all 
Army flight students. Their ages ranged from 18 to 39 years with a mean of 20.4 years, 
The class was divided into three flight sections; each section originally contained four 
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squads. For both the CEF and PAR administrations, a peer group consisted of one squad, 
with a mean N of nine candidates. The members of each squad were in their fourth week 
of preflight training at the time of the first CEF administration, and were in their fifth 
training week (first flight training week) when the three PAR forms were administered. 

The three PAR forms and the CFF axe contained in the Appendix. PAR Form 1 
contains a ranking, rating, and training success dichotomy answer sheet From an 
alphabetized list of his squad members' names, each student first cro^'*- 1 off his own 
name, then ranked his squadmates by writing their names in blanks proviu^d in order of 
“best potential aviation warrant officer" to “worst potential aviation warrent officer.” As 
the second step, each student rated his squadmates on a seven-point scale, using “1” for 
“best potential aviation warrant officer' and “7” for “wcrst.” 

Each of the seven ratings is composed of a one-line title, which is a general 
description, and three more specific descriptor sentences. These descriptors were devel- 
oped, in part, from critical incident descriptions during research previously reported by 
Boyles, Prunkl, and Wahlberg (7) concerned with development of a combat aviator 
criterion and, in part, from Behavior and Systems Research Laboratory's basic training 
peer rating scale, the Leadership Potential Rating (LPR). 

In addition to ranking and rating his peers, for each of his squadmates, each student 
answered “yes” or “no” to the question, “Do you think this man will complete flight 
training?” 

PAR Form 2 is identical to PAR 1 except that the PAR 2 ratings are shortened, 
being composed only of the one-line “title” descriptors. PAR Form 3 is a shorter, 
one-page form, on which the student need not rewrite his squadmates' names to rank 
them. After each squadmate’s name in the alphabetical listing, a numerical (1) rank and 
(2) rating from the scale (identical to the :cale on PAR Form 2) is assigned, and (3) the 
question of whether each squadmate will complete training is answered. 

The CEF presently in operational use yields (a) a ranking score, (b) a four-point 
rating for each of the 19 characteristics or tnuts shown on the form, and (c) a narrative 
“description of performance.” A CEF on each of his squadmates is completed by every 
trainee. Thus far, only CEF mean ranks have been compared with PAR scores. 

The basic comparison between PAR and CEF was aimed at discovering whether 
differences in validity existed between the absolute scale and ranks. An objective in 
administering the PARs was to develop a short, simple form amenable to group adminis- 
tration. CEFs are given to the members of a company with instructions to complete them 
in their quarters within a co pie of days. 

The CEF is administered during the 4th, 12th, and 16th weeks of training. To 
parallel these administrations as closely as possible, the PARs were administered during 
the 5th, 13th, and 17th weeks, four squads receiving each form. It was found that each 
of the three PAR forms could easily be administered during a half-hour group session. No 
significant differences existed between the times taken to complete PAR Form 1 and 
PAR Form 2. More errors were made by respondents on PAR Form 3, both in terms of 
inadvertent duplication of ranks and omission of ranks. Because of these findings, PAR 
Form 3 was dropped from further use. Since times to complete PAR Forms 1 and 2 were 
equal, it was decided to administer only PAR Form 1 on the second and third 
administrations. 

Table 1 shows correlations between the ratings and ranking on the PAR experi- 
mental forms, and between these and the rankings from the CEF. They are generally 
quite high. These values are acceptable even as coefficients or reliability. However, 
sufficient difference exists to make determination of relative validity of interest. 
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Table 1 



First Administration of PAR Experimental Forms 



PAR Ratify vs. PAR Ranking 


PAR Rating vs CEF 


PAR Ranking vs. CEF 


Form 1 - .04 
Form 2 = .95 
Form 3 = ,93 
Total = .92 


Form 1 - .87 
Form 2 = .91 
Form 3 - .84 
Total = .81 


Form 1 “ .91 
Form 2 - ,90 
Form 3 - .83 
Total - .86 



In Table 2, the data from the first administration are compared with the complete/ 
attrile criterion through the completion of primary helicopter training (20 weeks). The 
CEF “groups’' shown in this table are composed of the CEF scores cf those students 
receiving the respective experimental form of the PAR. The PAR Groups were groups of 
opportunity. 

The first four squads tested were given PAR Form 1, the second group of four 
squads Form 2, and the last group Form 3. Point biserial correlations and their bisenal 
equivalents are shown for each of five scorings of the PAR> mean ranks > mean ratings, 
and frequency of negative responses to the “will he complete” question were obtained as 
a first step. A mean for ranking-plus-rating was also computed, to derive « fourth score. 
To this score was added the number of “no” responses to the question that asks wheiner 
squadmates will complete flight training, to derive a fifth PAR score, which considers 
mean ranks, mean ratings, and frequency of “no’s.” The last two scoring methods are 
purely arbitrary. Additional scoring methods will be studied in the future. 

Within groups there is very little difference among CEF and PAR coefficients. All 
total PAR and CEF validity coefficients were significant at the .01 level. 

Because of the small subgroup Ns, the slight differences in alidity coefficients, and 
the surprising differences in attrition among groups, we cannot yet draw conclusions on 
comparative predictive strtngth. A chi-square computed among the three groups’ attrition 
data yielded a value of 10.4 (df = 2), which is significant at the .01 level. 

Table 3 also consists of data from the first administration. However, the criterion 
evidenced in this table is pass/fail through the 20 weeks of primary training rather than 
complete/attrite. A pass/fail criterion includes only those individuals who fail the program 
because of inadequate flight, academic, or military performance. Deleted from this type 
of analysis are those students who did not complete the course because of factors other 
than poor performance, such as medical problems, voluntary resignation, or personal 
problems. 

It was expected that the validity coefficients would increase when tne sample was 
limited to students who actually failed the program because of a flight, military, or 
academic deficiency. A comparison of Tables 2 and 3 will show that this expectation was 
not confirmed. 

In most leases for Group 1 (PAR Form 1) and Group 2 (PAR Form 2), the validity 
coefficients decreased. In Group 3 (PAR Form 3) the validity coefficients increased 
considerably. 

We believe the difference in the coefficients among PAR groups is a function of the 
differences in attrition >etween these groups. A chi-square among the pass/faiJ criterion 
groups yielded a value of 6.96 {df = 2, p<.10). 
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The attrition rate ranged from 33.3% for the group that received Form 1 to 71.4% 
for the group that received Form 3. This difference in attrition among the groups was 
not expected. In the recent past, flight classes have produced a mecji attrition rate of 
approximately 20%. For the first quarter of fiscal year 1969, for example, the attrition 
rate for classes ranged from 12% to 26%. Thus, for this class, and specifically for Group 
3, it was easier to predict attrition because it was more prevalent in this group than in 
any other previous sample. 

We have been cognizant of the fact that the:a are situational variables within certain 
companies and squads that would yield higher attrition rates within these groups. PAR 
Group 3 is an example of this situation. Ten of the 15 candidates who resigned from this 
class were initially in this group which produced the 71% attrition rate. (One candidate 
resigned from PAR Group 1 and four candidates resigned from PAR Group 3.) 

The fact that this situation occurred fcr the present sample, though unexpected, was 
not extremely surprising. It did, however, aifect the validity coefficients for these groups. 
It could be assumed that th*s situational variable, in turn, affected morale, and thus 
produced a higher failure rate within this group. 

As this class progresses through training, more attrition data will become available. 
On the basis of the absolute difference between PAR validity coefficients and those of 
the CEF, we will administer the PAR to additional classes for cross validation. Statistical 
comparisons between values of point biserial correlations tre risky. The appropriate test 
of comparative usefulness of the relatively and absolutely scaled forms will have to be the 
stability, as well as the relative size, of their validity coefficients. 

The original validity coefficients are sufficiently high to cause us to anticipate that 
the use of peer ratings may increase predictive accuracy in the multivariate system which 
Boyles has discussed, 1 The biserial equivalents of the total scores are, in most cases, 
above .40. These values compare quite favorably with the validity coefficients of other 
predictor variables available prior to actual training in the aircraft. 

Our goal is the earliest possible identification of poor risks in aviation training. Peer 
ratings are expected to increase the probabilit> of achieving that goal. 



'Wiley R. Boyle* and June* L. Wahlberg. “Prediction of Army Aviator Performance," HumRRO 
Profesaional Piper in preparation. 




6 



in 



LITERATURE CITED 
AND 

APPENDIX 




11 



LITERATURE CITED 



1. Weitz, J. “Selecting Supervisors With Peer Rating,” Personnel Psychol., vol. 11, 1968, 
pp. 25*35. 

2. Roadman, Harry E. “An Industrial Use of Peer Ratings,” J . Appl. Psychol ., vol. 48, 
1964, pp.’ 211-214. 

3. Waters, L.K., and Waters, C.W. “Peer Nominations as Predictors of Short-Term Sales 
Performance,” J. Appl, Psychol. , vol. 54, 1970, pp. 42-44. 

4. McClure, Glenn E., Tupes, Ernest C., and Dailey, John T. Research on C™*™ °f 
Officer Effectiveness , Air Training Command Research Bulletin 51-8, May 1951. 

5. Hollander, E.P. The Validity of Peer Ratings from OCS in Predicting Officer Per- 
formance: A Follow-up of the 1955 Newport Study , Office of Naval Research 
Technical Report, May 1964. 

6. Medland, Francis F. Research to Identify Potential NCO Leaders , Army Personnel 
Research Office Research Study 64-1, May 1964. 

7. Boyles, Wiley R., Prunkl, Peter R., and Wahlberg, James L. Combat Aiiator Criterion 
Development , HumRRO Professional Pap* 34-09, November 1969. 



Appendix 

POTENTIAL AVIATION WARRANT OFFICER RATING FORM 



This is an opportunity to use some of the skills you may need later as an Aviation Warrant 
Officer. In the future you may have the duty of evaluating other aviaton, and subordinates. 
Today you are goin^ to evaluate your fellow squad members in terms of how you believe 
they will pei^rm later as Aviation Warrant Officers. 

We urge you to be completely honest and to follow instructions carefully as the purpose of 
this rating form is to determine whether >our ratings will be a useful addition in developing 
a predictor for selection of future candidates. 

This rating form is for OFFICIAL RESEARCH PURPOSES ONLY and WILL NOT be phced 
in your personnel records or be made available to your commanders or supervisors. 

We welcome any comment on the format or text that can be used in improving riie*e rat- 
ings. Feel free to write your comments on any available blank space in this booklet. 

This “Peer Rating” is concerned with how you think your felicw squad members will perform 
as Aviation Warrant Officers. Your evaluations will be held in strict confidence and will be 
seen by research personnel only. 
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INSTRUCTIONS 

Complete each of the following steps carefully. 

A. BACKGROUND INFORMATION : 

(1) PULL OUT THE ANSWER SHEET AND TURN IT TO SIDE 1. FILL IN ALL THE 
INFORMATION ON THIS SIDE. 

(2) TURN THE ANSWER SHEET OVER TO SIDE 2. 

B. RANKING : 

(1) DRAW A LINE THROUGH YOUR OWN NAME— DO NOT RANK YOURSELF. 

(2) STUDY THE REMAINING LIST OF NAMES AND DECIDE WHO WILL BE THE 
BEST AVIATION WARRANT OFFICER. WRITE HIS NAME IN THE TOP BLANK 
DESIGNATED “BEST,” THEN CROSS HIS NAME OFF THE LIST. 

13) STUDY THE REMAINING LIST OF NAMES AND DECIDE WHO WILL BE THE 
WORST AVIATION WARRANT OFFICER. WRITE HIS NAME IN THF BOTTOM 
BLANK DESIGNATED “WORST,” THEN CROSS HIS NAME OFF THE LIST. 

(4) PICK FROM THE REMAINING LIST OF NAMES THE ONE WHO WILL BE THE 
BEST AVIATION WARRANT OFFICER. WRITE HIS NAME IN THE “NEXT 
BEST” BLANK, THEN CROSS HIS NAME OFF THE LIST. 

(5) FROM THE REMAINING NAMES, PICK THE ONE WHO WILL BE THE WORST 
AVIATION WARRANT OFFICER, PUT HIS NAME IN THE “NEXT WORST” 
BLANK, THEN CROSS HIS NAME OFF THE LIST. 

(6) CONTINUE TO CHOOSE THE BEST AND WORST POTENTIAL AVIATION 
WARRANT OFFICERS UNTIL ALL THE NAMES ARE CROSSED OUT. WHEN 
YOU ARE THROUGH, YOU SHOULD HAVE EVERY MEMBER OF YOUR SQUAD 
RANKED FROM BEST TO WORST. IF THERE ARE FEWER THAN i5 MEN IN 
YOUR SQUAD, THERE SHOULD BE BLANK SPACES IN THE MIDDLE OF THE 
RANKING. 

C. RATING : 

Study the descriptions of Aviation Warrants on page 3. Use these descriptions to 
rate each of the men you have Just ranked. Your “Best” choice does not necessarily 
have- to be rated “1,” neither does your "Worst” have to be rated “7.” Use the num- 
ber for each man which you feel best describes what kind of Aviation \\airant Officer 
he really will be. 

Rate your squadmates in the same sequence that you ranked them. First, rate 
your “Best” choice, then rate your “Worst,” then “Next Best,” then “Next Worst,” 
and so on, until you have assigned description numbers to all the men in your squad, 
Do not give a man a description number higher than that of a man you ranked 
above him. Do not give a man a description number lower than that of a man you 
ranked below him. Use 8ny description numbers as many times as you think 
appropriate. 

(1) NOW, PUT THE MOST APPROPRIATE DESCRIPTION NUMBER IN FACH OF THE 
BOXES NEXT TO THE NAMES OF EACH OF THE MEN YOU HAVE RANKED. 

(2) FINALLY, PUT “YES” OR "NO” FOR EACH MAN IN THE COLUMN WHICH ASKS 
“DO YOU THINK THIS MAN WILL COMPLETE FLIGHT TRAINING?" 

After you have completed every step on this page, sign your initial? here , and 

close the booklet. 

Thank you for your time and effort on this rating form. 




Potential Aviation Warrant Officer Descriptions 



1 


THE VERY BEST TYPE OF AVIATION WARRANT OFFICER 

I would stake my life on him to know his job and do it right at all times. 
He will keep a clear head in any situation. 

Men will be eager to work with him and for him. 




2 


AN EXTREMELY HIGH TYPE OF AVIATION WARRANT OFFICER 
He will have very good judgment. 

He will do a great deal more than what is expected of him. 

He vill adjust to any changing situation. 






3 


A VERY GOOD AVIATION WARRANT OFFICER 

He will have good judgment. 

He will remain calm and relaxed in any situation. 

He will have what it takes to do the job right. 




4 


A GOOD AVIATION WARRANT OFFICER 
His judgment will be average. 

His decisions may not always be correct, but he will usually come through. 
He will do an average job. 




5 


A FAIR AVIATION WARRANT OFFICER 

He will get nervous in tough situations, 

His judgment will be below average. 

He may do his job, but nothing more. 




6 


A POOR AVIATION WARRANT OFFICER 
He will lose his head in tight situations. 

He will have very little of what it takes to be an Aviation Warrant Officer. 
He will usually fail to get his job done. 




7 


THE WORST TYPE OF AVIATION WARRANT OFFICER 


He will panic in tough situations. 

Men will refuse to work with him. 

He will have the worst judgment of anyone in his unit. 



0 




ANSWER SHEET 



Side 1 

1. Your name, 2> SSAN 

Last First MI 

3. Age 4. Today’s Date 1 1 5. Class Number 

Month Day Year 

6. Section Number 7. Squad Number 

8. How many weeks have you been a member of this squad? 

01234667 
(Circle one) 

9. Your present position in this squad: 

(squad leader, squad member, section leader, etc.) 



When you have completed all items on this page, sign your initials here 
then fill out the rest of this booklet. 
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ANSWER SHEET 
Side 2 

Alphabetical Listing of Squad Members' Names 
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CONFIDENTIAL: To Be Seen By Research Personnel Only 



You will be asked to make several judgments about each of the men whose 
names are listed below, except yourself. Forget about who you like or dislike — 
and make your judgments on the basis of what you actually think about each of 
these men. 

Step 1. Draw a line through your own name and do not rate yourself at all. 

Step 2. RANKING Column #1: On the basis of what kinds of Aviation Warrant Officers 
you think they will be, rank these men from BEST to WORST. (1 = Best, 

2 = Next Best, . . , the highest number will be assigned to the man you think 
will be the Worst Aviation Warrant Officer.) 

Step 3. DESCRIPTION Column #2: Read the following descriptions. Assign each man 
the nur.v er of the description you think best /its the kind of Aviation Warrant 
Off ; cer he will be. 

1. THE VERY BEST TYPE OF AVIATION WARRANT OFFICER. 

2. AN EXTREMELY HIGH TYPE OF AVIATION WARRANT OFFICER. 

3. A VERY GOOD AVIATION WARRANT OFFICER. 

4. A GOOD AVIATION WARRANT OFFICEF. 

5. A FAIR AVIATION WARRANT OFFICER 

6. A POOR AVIATION WARRANT OFFICER. 

7. THE WORST TYPE OF AVIATION WARRANT OFFICER. 



Step 4. COLUMN 3: Answer “yes” or “no” to the question in this column for each 
man listed. 



Alphabetic Listing of Names 


RANKING 

1 


DESCRIPTION 

2 


Will he 
complete 
flight 
training? 

3 


. 
















V ^ 




N 



1. 


Your name 




2. 


SSAN 


3. 


Last First 

Age 4. Today’s Date 


MI 


5. 


Class Number 


6. 


Month 

Section Number 


Day Year 


7. 


Squad Number. 



8. How many weeks have you been a member of this, squad? 



0 1 2 3 4 6 6 7 
(Circle one) 

9. Your present position in this squad: 




(squad leader, squad member, section leader, etc.) 



15 

! 18 



CONTEMPORARY EVALUATION FORM 



Name of Rated Candidate (Last, First, MI) 



Section 



Period of Evaluation 



Date 



KEY TO EVALUATION RATING 



Unsatisfactory 

U 


Below Average 
BA 


Average 

A 


Above Average 
AA 


Rating 


Description 


Rating 




Description 


Rating 


Description 


A. 


ADAPTABILITY 




0. 


EXPRESSION 


M. 


LOYALTY 


B. 


APPEARANCE 




H 


FORCE 


N. 


MORAL COURAGE 


C. 


ATTENTION 




I 


INGENUITY 


0. 


MOTIVATION 


D. 


COOPERATION 




J. 


INITIA llVE 


P. 


SELF-DISCIPLINE 


E. 


DEPENDABILITY 


K. 


INTELLIGENCE 


Q 


STAMINA 


__ F. 


ENTHUSIASM 




L. 


JUDGMENT 


R. 


TACT 






s. 


UNDERSTANDING 



DESCRIPTION OF PERFORMANCE 




o 

ERJC 

\ 



RATED. 



.OF. 



SIGNATURE 



(PRINT) RATING PERSON 
USAPHC/S Form 441 (Rev) (HTB) 18 June 1969 



TAG OFFICER’S INITIALS, 

f 



16 



in 



ERIC -house 



MAY 1 T 1371 

on Atluu uviiialioo 



